Predicting structured illumination parameters

ABSTRACT

Implementations of the disclosure are directed to predicting structured illumination parameters for a particular point in time, space, and/or temperature using estimates of structured illumination parameters obtained from structured illumination images captured by a structured illumination system. Particular implementations are directed to predicting structured illumination frequency, phase, orientation, and/or modulation order parameters.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 17/090,775, filed on Nov. 5, 2020 and issued as U.S. Pat. No.11,119,302 on Sep. 14, 2021, titled “PREDICTING STRUCTURED ILLUMINATIONPARAMETERS”, which is a continuation of U.S. patent application Ser. No.16/447,729, filed on Jun. 20, 2019 and issued as U.S. Pat. No.10,831,012 on Nov. 10, 2020, titled “PREDICTING STRUCTURED ILLUMINATIONPARAMETERS”, which claims the benefit of U.S. Provisional PatentApplication No. 62/692,303 filed on Jun. 29, 2018 and titled “PREDICTINGSTRUCTURED ILLUMINATION PARAMETERS.”

BACKGROUND

Structured illumination microscopy (SIM) describes a technique by whichspatially structured (i.e., patterned) light may be used to image asample to increase the lateral resolution of the microscope by a factorof two or more. In some instances, during imaging of the sample, threeimages of fringe patterns of the sample are acquired at various patternphases (e.g., 0°, 120°, and 240°), so that each location on the sampleis exposed to a range of illumination intensities, with the procedurerepeated by rotating the pattern orientation about the optical axis to 3separate angles (e.g. 0°, 60° and 120°). The captured images (e.g., nineimages) may be assembled into a single image having an extended spatialfrequency bandwidth, which may be retransformed into real space togenerate an image having a higher resolution than one captured by aconventional microscope.

In some implementations of current SIM systems, a linearly polarizedlight beam is directed through an optical beam splitter that splits thebeam into two or more separate orders that may be combined and projectedon the imaged sample as an interference fringe pattern with a sinusoidalintensity variation. Diffraction gratings are examples of beam splittersthat can generate beams with a high degree of coherence and stablepropagation angles. When two such beams are combined, the interferencebetween them can create a uniform, regularly-repeating fringe patternwhere the spacing is determined by factors including the angle betweenthe interfering beams.

During capture and/or subsequent assembly or reconstruction of imagesinto a single image having an extended spatial frequency bandwidth, thefollowing structured illumination parameters may need to be considered:the spacing between adjacent fringes (i.e., frequency of fringepattern), the phase or angle of the structured illumination pattern, andthe orientation of the fringe pattern relative to the illuminatedsample. In an ideal imaging system, not subject to factors such asmechanical instability and thermal variations, each of these parameterswould not drift or otherwise change over time, and the precise SIMfrequency, phase, and orientation parameters associated with a givenimage sample would be known. However, due to factors such as mechanicalinstability of an excitation beam path and/or thermalexpansion/contraction of an imaged sample, these parameters may drift orotherwise change over time.

As such, a SIM imaging system may need to estimate structuredillumination parameters to account for their variance over time. As manySIM imaging systems do not perform SIM image processing in real-time(e.g., they process captured images offline), such SIM systems may spenda considerable amount of computational time to process a SIM image toestimate structured illumination parameters for that image.

SUMMARY

Implementations of the disclosure are directed to predicting structuredillumination parameters for a particular point in time, space, and/ortemperature using estimates of structured illumination parametersobtained from structured illumination images captured by a structuredillumination system.

In one example, a method comprises: using a structured illuminationsystem to capture a first image of a sample; using a computing device toestimate a first structured illumination parameter using at least thecaptured first image; using the structured illumination system tocapture a second image of the sample; using the computing device toestimate a second structured illumination parameter using at least thecaptured second image; and using at least the first structuredillumination parameter or the second structured illumination parameter,using the computing device to predict a third structured illuminationparameter corresponding to a third image. Each of the first, second, andthird structured illumination parameters may comprise a phase,frequency, orientation, or modulation order.

In some implementations, the first image is captured at a first time,the second image is captured at a second time after the first time, thethird image is captured at a third time between the first time and thesecond time, and the third structured illumination parameter ispredicted at the third time by using at least an interpolation method.The interpolation method may comprise: using the computing device todetermine a rate of change from the first structured illuminationparameter at the first time to the second structured illumination at thesecond time; and using at least the determined rate of change, using thecomputing device to predict the third structured illumination parameterat the third time.

In some implementations, the method further comprises: using thecomputing device to construct a high resolution image using at least thethird image and the third structured illumination parameter.

In some implementations: the first image is captured at a first time,the second image is captured at a second time after the first time, thethird image is captured at a third time after or before both the firsttime and the second time, and the third structured illuminationparameter is predicted at the third time by using at least anextrapolation method.

In some implementations, the method further comprises: using at leastthe third structured illumination parameter to adjust a hardwarecomponent of the structured illumination system to compensate forchanges in a structured illumination parameter prior to capturing thethird image at the third time. Adjusting a hardware component maycomprise: adjusting one or more of: a rotating mirror to adjust a phaseor orientation of a structured illumination pattern, a translation stagecarrying a diffraction grating to adjust a phase or orientation of astructured illumination pattern, and a sample translation stage toadjust a phase or orientation of a structured illumination pattern.

In some implementations, the method further comprises: storing in amemory of the structured illumination system: the first structuredillumination parameter, the second structured illumination parameter,and the third structured illumination parameter; and using one or moreof the stored first structured illumination, stored second structuredillumination parameter, stored third structured illumination parameter,and a stored value based on the known physical characteristics of thestructured illumination system to reduce a search space fora fourthstructured illumination parameter fora fourth image.

In some implementations, predicting the third structured illuminationparameter corresponding to the third image comprises: applying aleast-squares fit to at least the first structured illuminationparameter and the second structured illumination parameter. In someimplementations, predicting the third structured illumination parametercorresponding to the third image comprises: using the second structuredillumination parameter.

In some implementations, the first image of the sample is captured at afirst sample temperature; the first structured illumination parameter isestimated at the first sample temperature; the second image of thesample is captured at a second sample temperature; the second structuredillumination parameter is estimated at the second sample temperature;and the third structured illumination parameter is predicted at a thirdsample temperature.

In some implementations, the method further comprises: dividing thefirst image of the sample into a plurality of image subsections; usingthe computing device to estimate a fourth structured illuminationparameter using at least a first image subsection of the plurality ofimage subsections; using the computing device to estimate a fifthstructured illumination parameter using at least a second imagesubsection of the plurality of image subsections; using at least thefourth structured illumination parameter or the fifth structuredillumination parameter, using the computing device to predict a sixthstructured illumination parameter corresponding to a third imagesubsection of the plurality of image subsections.

In some implementations, the method further comprises: dividing thefirst image of the sample into a plurality of image subsections; usingthe computing device to estimate a fourth structured illuminationparameter using at least a first image subsection of the plurality ofimage subsections; and using the estimated fourth structuredillumination parameter as a predicted structured illumination parameterof a second image subsection of the plurality of image subsections.

In one example, a non-transitory computer-readable medium may haveexecutable instructions stored thereon that, when executed by aprocessor, cause the processor to perform operations of: using astructured illumination system to capture a first image of a sample;estimating a first structured illumination parameter using at least thecaptured first image; using the structured illumination system tocapture a second image of the sample; estimating a second structuredillumination parameter using at least the captured second image; andusing at least the first structured illumination parameter or the secondstructured illumination parameter, predicting a third structuredillumination parameter corresponding to a third image.

In some implementations, the first image is captured at a first sampleposition, the second image is captured at a second sample position, thethird image is captured at a third sample position between the firstsample position and the second sample position, and the third structuredillumination parameter is predicted at the third sample position byusing at least an interpolation method. The interpolation method maycomprise: using the computing device to determine a rate of change fromthe first structured illumination parameter at the first sample positionto the second structured illumination at the second sample position; andusing at least the determined rate of change, predicting the thirdstructured illumination parameter at the third sample position.

In some implementations, the instructions when executed by theprocessor, cause the processor to further perform an operation of:constructing a high resolution image using at least the third image andthe third structured illumination parameter.

In some implementations, the third sample position is after the firstsample position and the second sample position, and the third structuredillumination parameter is predicted at the third sample position byusing at least an extrapolation method.

In some implementations, the instructions when executed by theprocessor, cause the processor to further perform an operation of: usingat least the third structured illumination parameter to cause a hardwarecomponent of the structured illumination system to be adjusted tocompensate for changes in a structured illumination parameter prior tocapturing an image at the third sample position.

In some implementations, the instructions when executed by theprocessor, cause the processor to further perform operations of: storingin a memory of the structured illumination system: the first structuredillumination parameter, the second structured illumination parameter,and the third structured illumination parameter; and using one or moreof the stored first structured illumination, stored second structuredillumination parameter, stored third structured illumination parameter,and a stored value based on the known physical characteristics of thestructured illumination system to reduce a search space for a fourthstructured illumination parameter for a fourth image.

In one example, a structured illumination imaging system comprises: alight emitter to emit light; a beam splitter to split light emitted bythe light emitter to project a structured illumination pattern on aplane of a sample; a processor; and a non-transitory computer-readablemedium having executable instructions stored thereon that, when executedby the processor, cause the processor to perform operations of:capturing a first image of a sample; estimating a first structuredillumination parameter using at least the captured first image;capturing a second image of the sample; estimating a second structuredillumination parameter using at least the captured second image; andusing at least the first structured illumination parameter or the secondstructured illumination parameter, predicting a third structuredillumination parameter corresponding to a third image.

In one example, a method comprises: using a structured illuminationsystem to capture a first plurality of images of a sample; using acomputing device to estimate a first structured illumination parameterusing at least the captured first plurality of images; using thestructured illumination system to capture a second plurality of imagesof the sample; using the computing device to estimate a secondstructured illumination parameter using at least the captured secondplurality of images; and using at least the first structuredillumination parameter or the second structured illumination parameter,using the computing device to predict a third structured illuminationparameter corresponding to one or more images.

Other features and aspects of the disclosed technology will becomeapparent from the following detailed description, taken in conjunctionwith the accompanying drawings, which illustrate, by way of example, thefeatures in accordance with implementations of the disclosed technology.The summary is not intended to limit the scope of any inventionsdescribed herein, which are defined by the claims and equivalents.

It should be appreciated that all combinations of the foregoing concepts(provided such concepts are not mutually inconsistent) are contemplatedas being part of the inventive subject matter disclosed herein. Inparticular, all combinations of claimed subject matter appearing at theend of this disclosure are contemplated as being part of the inventivesubject matter disclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure, in accordance with one or more implementations,is described in detail with reference to the following figures. Thefigures are provided for purposes of illustration only and merely depictexample implementations. Furthermore, it should be noted that forclarity and ease of illustration, the elements in the figures have notnecessarily been drawn to scale.

Some of the figures included herein illustrate various implementationsof the disclosed technology from different viewing angles. Although theaccompanying descriptive text may refer to such views as “top,” “bottom”or “side” views, such references are merely descriptive and do not implyor require that the disclosed technology be implemented or used in aparticular spatial orientation unless explicitly stated otherwise.

FIG. 1A illustrates, in one example, undesired changes in frequency thatmay occur over time in a SIM imaging system that projects a 1Dstructured illumination pattern on a sample.

FIG. 1B illustrates, in one example, undesired changes in phase that mayoccur over time in a SIM imaging system that projects a 1D structuredillumination pattern on a sample.

FIG. 1C illustrates, in one example, undesired changes to orientationthat may occur over time in a SIM imaging system that projects a 1Dstructured illumination pattern on a sample.

FIG. 2 illustrates, in one example, a SIM imaging system that mayimplement structured illumination parameter prediction in accordancewith some implementations described herein.

FIG. 3 is an optical diagram illustrating an example opticalconfiguration of a two-arm SIM imaging system that may implementstructured illumination parameter prediction in accordance with someimplementations described herein.

FIG. 4 illustrates, in one example, simplified illumination fringepatterns that may be projected onto the plane of a sample by a verticalgrating and horizontal grating of the SIM imaging system of FIG. 3during one imaging cycle to use structured light to create ahigh-resolution image.

FIG. 5A is a schematic diagram illustrating an example opticalconfiguration of a dual optical grating slide SIM imaging system thatmay implement structured illumination parameter prediction, inaccordance with some implementations described herein.

FIG. 5B is a schematic diagram illustrating an example opticalconfiguration of a dual optical grating slide SIM imaging system thatmay implement structured illumination parameter prediction, inaccordance with some implementations described herein.

FIG. 6 illustrates, in one example, simplified illumination fringepatterns that may be projected onto the plane of a sample by a firstdiffraction grating and a second diffraction grating of the SIM imagingsystem of FIGS. 5A-5B during image capture for a structured illuminationimaging cycle.

FIG. 7 shows, in one example, an estimate of a phase parameter thatvaries in space (X) and time (T).

FIG. 8 illustrates, in one example, a trend of the estimate variation ofa parameter as a function of x.

FIG. 9 is an operational flow diagram illustrating an exampleinterpolation method for predicting structured illumination parametersfor a particular point in time using estimates of structuredillumination parameters obtained from images captured before and afterthe point time, in accordance with some implementations describedherein.

FIG. 10 is an operational flow diagram illustrating an exampleextrapolation method for predicting structured illumination parametersfor a particular point in time using estimates of structuredillumination parameters obtained from two or more images captured beforethe point in time, in accordance with some implementations describedherein.

FIG. 11 is an operational flow diagram illustrating an example method ofusing a predicted structured illumination parameter during highresolution image reconstruction to compensate for undesired changes instructured illumination parameters over time, in accordance with someimplementations described herein.

FIG. 12 is an operational flow diagram illustrating an example method ofusing a predicted structured illumination parameter adjustments of SIMimaging system hardware components to compensate for structuredillumination parameter changes over time, in accordance with someimplementations described herein.

FIG. 13 is an example of a computing component that can be used inconjunction with various implementations of the present disclosure.

The figures are not exhaustive and do not limit the present disclosureto the precise form disclosed.

DETAILED DESCRIPTION

As used herein to refer to a structured illumination parameter, the term“frequency” is intended to refer to a spacing between fringes or linesof a structured illumination pattern (e.g., fringe or grid pattern). Forexample, a pattern having a greater spacing between fringes will have alower frequency than a pattern having a lower spacing between fringes.

As used herein to refer to a structured illumination parameter, the term“phase” is intended to refer to a phase of a structured illuminationpattern illuminating a sample. For example, a phase may be changed bytranslating a structured illumination pattern relative to an illuminatedsample.

As used herein to refer to a structured illumination parameter, the term“orientation” is intended to refer to a relative orientation between astructured illumination pattern (e.g., fringe or grid pattern) and asample illuminated by the pattern. For example, an orientation may bechanged by rotating a structured illumination pattern relative to anilluminated sample.

As used herein to refer to a structured illumination parameter, theterms “predict” or “predicting” are intended to mean calculating thevalue(s) of the parameter without directly measuring the parameter orestimating the parameter from a captured image corresponding to theparameter. For example, a phase of a structured illumination pattern maybe predicted at a time t1 by interpolation between phase values directlymeasured or estimated (e.g., from captured phase images) at times t2 andt3 where t2<t1<t3. As another example, a frequency of a structuredillumination pattern may be predicted at a time t1 by extrapolation fromfrequency values directly measured or estimated (e.g., from capturedphase images) at times t2 and t3 where t2<t3<t1.

As used herein to refer to light diffracted by a diffraction grating,the term “order” or “order number” is intended to mean the number ofinteger wavelengths that represents the path length difference of lightfrom adjacent slits or structures of the diffraction grating forconstructive interference. The interaction of an incident light beam ona repeating series of grating structures or other beam splittingstructures can redirect or diffract portions of the light beam intopredictable angular directions from the original beam. The term “zerothorder” or “zeroth order maximum” is intended to refer to the centralbright fringe emitted by a diffraction grating in which there is nodiffraction. The term “first-order” is intended to refer to the twobright fringes diffracted to either side of the zeroth order fringe,where the path length difference is ±1 wavelengths. Higher orders arediffracted into larger angles from the original beam. The properties ofthe grating can be manipulated to control how much of the beam intensityis directed into various orders. For example, a phase grating can befabricated to maximize the transmission of the ±1 orders and minimizethe transmission of the zeroth order beam.

As used herein to refer to a sample, the term “feature” is intended tomean a point or area in a pattern that can be distinguished from otherpoints or areas according to relative location. An individual featurecan include one or more molecules of a particular type. For example, afeature can include a single target nucleic acid molecule having aparticular sequence or a feature can include several nucleic acidmolecules having the same sequence (and/or complementary sequence,thereof).

As used herein, the term “xy plane” is intended to mean a 2-dimensionalarea defined by straight line axes x and y in a Cartesian coordinatesystem. When used in reference to a detector and an object observed bythe detector, the area can be further specified as being orthogonal tothe beam axis, or the direction of observation between the detector andobject being detected.

As used herein, the term “z coordinate” is intended to mean informationthat specifies the location of a point, line or area along an axis thatis orthogonal to an xy plane. In particular implementations, the z axisis orthogonal to an area of an object that is observed by a detector.For example, the direction of focus for an optical system may bespecified along the z axis.

As used herein, the term “optically coupled” is intended to refer to oneelement being adapted to impart light to another element directly orindirectly.

As noted above, parameter estimation for SIM image processing may beneeded to correct for undesired changes in structured illuminationparameters over time. By way of example, FIGS. 1A-1C illustrateundesired changes in frequency (FIG. 1A), phase (FIG. 1B), andorientation (FIG. 1C) that may occur over time in a SIM imaging systemthat projects a one-dimensional structured illumination pattern on aregularly patterned sample. In particular, FIG. 1A illustrates a sample50 with features 51 illuminated by a one-dimensional structuredillumination pattern having fringes 60, before and after frequencyshifts. Before any frequency shifts, adjacent fringes 60 have a pitch orcenter-to-center-spacing of P corresponding to an initial frequency f.Over time, with temperature variations in the system, the pitch P mayincrease or decrease. For example, thermal expansion may cause the pitchP to increase to P+ΔP₁, correspondingly decreasing the frequency f tof−Δf₁. Conversely, thermal contraction may cause the pitch P to decreaseto P−ΔP₁, correspondingly increasing the frequency f to f+Δf₂.

FIG. 1B illustrates the sample 50 illuminated by a one-dimensionalstructured illumination pattern having fringes 60, before and afterchanges in a phase. As shown, before phase drift, a first phase state ϕmay correspond to fringes completely illuminating every second column offeatures 51 of sample 50. Over time, the position of the fringes 60relative to the sample 50 may shift such that all phase images areoffset by Δϕ. For example, mechanical vibrations in the SIM imagingsystem (e.g., in an excitation beam path), imprecision in a translationstage used by a grating or sample stage, thermal variations, and/orother factors may cause an undesired drift in the phase. After the phasedrifts by Δϕ, the first phase state changes to ϕ+Δϕ, and the fringes nolonger are centered on every second column of features.

FIG. 1C illustrates the sample 50 illuminated by a one-dimensionalstructured illumination pattern having fringes 60, before and afterchanges in orientation. As shown, before a change in orientation, theorientation of the fringes relatively to sample 50 are completelyvertical. Over the time, the orientation may change due to factors suchas changes in the excitation beam path, movement of the sample, thermalvariations, and/or other factors. After the orientation rotates by anangle Δθ, the fringes are no longer completely vertical relative to thesample.

Parameter estimation during an SIM imaging process to precisely accountfor changes in structured illumination parameters as described abovehelps ensure an artifact-free and accurate reconstruction of an imagefrom a set of sampled images. However, such a process may becomputationally expensive and is frequently performed after imageacquisition. For time-critical SIM imaging systems that involvereal-time processing and reconstruction of images, and thus real-timeestimation of parameters such as frequency, phase, orientation, andmodulation order, these computational requirements may result in a lossof data throughput (e.g., less data may be processed per unit of time).In such systems, the rate at which samples are imaged may exceed therate at which structured illumination parameters may be directlyestimated from the sampled images. As such, there is a need for a methodof generating a parameter estimate with low complexity and lowprocessing time.

To this end, implementations of the technology disclosed herein aredirected to predicting structured illumination parameters for aparticular point in time, space, and/or temperature using estimates ofstructured illumination parameters obtained from images captured by thestructured illumination system. Particular implementations are directedto predicting structured illumination frequency, phase, orientation,and/or modulation order parameters.

In accordance with some implementations, a structured illuminationparameter may be predicted for a given point in time, space, and/ortemperature by interpolating estimates of the structured illuminationparameter from image captures. For example, a first frequency may beestimated from a first sampled image, a second frequency may beestimated from a second sampled image, and a frequency corresponding toa point in time between the first captured image and the second capturedimage (e.g., a frequency for an image taken between the first and secondimages) may be predicted by interpolating using at least a determinedrate of change of the frequency between the first captured image and thesecond captured image.

In accordance with some implementations, a structured illuminationparameter may be predicted for a given point in time, space, and/ortemperature by extrapolation using estimates of a structuredillumination parameter obtained from two image captures. For example, afirst orientation may be estimated from a first sampled image, a secondorientation may be estimated from a second sampled image, and anorientation corresponding to a point in time after the first and secondcaptured images (e.g., an orientation for a third image taken after thefirst and second images) may be predicted by extrapolation using atleast a determined rate of change of the orientation from the firstcaptured image to the second captured image. As a second example, afirst orientation may be estimated from a first sampled image, a secondorientation may be estimated from a second sampled image, and anorientation corresponding to a point in time after the first and secondcaptured images (e.g., an orientation for a third image taken after thefirst and second images) may be predicted by holding the value from thesecond captured image.

In implementations, estimated and predicted structured illuminationparameters may be used to narrow a search space for other structuredillumination parameters that are predicted. For example, given anestimated value of a structured illumination parameter for a first pointin time, space, and/or temperature, a value of the structuredillumination parameter for second point in time, space, and/ortemperature that is near the first point in time, space, and/ortemperature may be predicted taking into account the predicted orestimated value at the first point in time, space, and/or temperature.

In implementations, estimated and predicted structured illuminationparameters may be stored in a memory of the structured illuminationsystem for later use by the system. For instance, predicted andestimated parameters may be stored in a history file such as a lookuptable. Predicted parameters that are stored in memory may be determinedfrom estimated parameters, or they may be set based on the physicalcharacteristics of the structured illumination system. For example, thenominal grid spacing of the structured illumination system may bestored. The stored parameters may thereafter be referenced to performoperations such as: calibrated image reconstruction, providing feedbackto a hardware component to correct for changes in structuredillumination parameters, and narrowing the search space when predictingadditional structured illumination parameters.

Before describing various implementations of techniques disclosed hereinfor predicting structured illumination parameters, it is useful todescribe example SIM imaging systems with which these techniques can beimplemented. FIGS. 2-6 illustrate three such example SIM imagingsystems. It should be noted that while these systems are describedprimarily in the context of SIM imaging systems that generate 1Dillumination patterns, the technology disclosed herein may implementedwith SIM imaging systems that generate higher dimensional illuminationpatterns (e.g., two-dimensional grid patterns).

FIG. 2 illustrates a structured illumination microscopy (SIM) imagingsystem 100 that may implement structured illumination parameterprediction in accordance with some implementations described herein. Forexample, system 100 may be a structured illumination fluorescencemicroscopy system that utilizes spatially structured excitation light toimage a biological sample.

In the example of FIG. 2 , a light emitter 150 is configured to output alight beam that is collimated by collimation lens 151. The collimatedlight is structured (patterned) by light structuring optical assembly155 and directed by dichroic mirror 160 through objective lens 142 ontoa sample of a sample container 110, which is positioned on a motionstage 170. In the case of a fluorescent sample, the sample fluoresces inresponse to the structured excitation light, and the resultant light iscollected by objective lens 142 and directed to an image sensor ofcamera system 140 to detect fluorescence.

Light structuring optical assembly 155 includes one or more opticaldiffraction gratings or other beam splitting elements (e.g., a beamsplitter cube or plate) to generate a pattern of light (e.g., fringes,typically sinusoidal) that is projected onto samples of a samplecontainer 110. The diffraction gratings may be one-dimensional ortwo-dimensional transmissive or reflective gratings. The diffractiongratings may be sinusoidal amplitude gratings or sinusoidal phasegratings.

In some implementations, the diffraction grating(s)s may not utilize arotation stage to change an orientation of a structured illuminationpattern. In other implementations, the diffraction grating(s) may bemounted on a rotation stage. In some implementations, the diffractiongratings may be fixed during operation of the imaging system (i.e., notrequire rotational or linear motion). For example, in a particularimplementation, further described below, the diffraction gratings mayinclude two fixed one-dimensional transmissive diffraction gratingsoriented perpendicular to each other (e.g., a horizontal diffractiongrating and vertical diffraction grating).

As illustrated in the example of FIG. 2 , light structuring opticalassembly 155 outputs the first orders of the diffracted light beams(e.g., m=±1 orders) while blocking or minimizing all other orders,including the zeroth orders. However, in alternative implementations,additional orders of light may be projected onto the sample.

During each imaging cycle, imaging system 100 utilizes light structuringoptical assembly 155 to acquire a plurality of images at various phases,with the fringe pattern displaced laterally in the modulation direction(e.g., in the x-y plane and perpendicular to the fringes), with thisprocedure repeated one or more times by rotating the pattern orientationabout the optical axis (i.e., with respect to the x-y plane of thesample). The captured images may then be computationally reconstructedto generate a higher resolution image (e.g., an image having about twicethe lateral spatial resolution of individual images).

In system 100, light emitter 150 may be an incoherent light emitter(e.g., emit light beams output by one or more excitation diodes), or acoherent light emitter such as emitter of light output by one or morelasers or laser diodes. As illustrated in the example of system 100,light emitter 150 includes an optical fiber 152 for guiding an opticalbeam to be output. However, other configurations of a light emitter 150may be used. In implementations utilizing structured illumination in amulti-channel imaging system (e.g., a multi-channel fluorescencemicroscope utilizing multiple wavelengths of light), optical fiber 152may optically couple to a plurality of different light sources (notshown), each light source emitting light of a different wavelength.Although system 100 is illustrated as having a single light emitter 150,in some implementations multiple light emitters 150 may be included. Forexample, multiple light emitters may be included in the case of astructured illumination imaging system that utilizes multiple arms,further discussed below.

In some implementations, system 100 may include a tube lens 156 that mayinclude a lens element to articulate along the z-axis to adjust thestructured beam shape and path. For example, a component of the tubelens may be articulated to account for a range of sample thicknesses(e.g., different cover glass thickness) of the sample in container 110.

In the example of system 100, fluid delivery module or device 190 maydirect the flow of reagents (e.g., fluorescently labeled nucleotides,buffers, enzymes, cleavage reagents, etc.) to (and through) samplecontainer 110 and waste valve 120. Sample container 110 can include oneor more substrates upon which the samples are provided. For example, inthe case of a system to analyze a large number of different nucleic acidsequences, sample container 110 can include one or more substrates onwhich nucleic acids to be sequenced are bound, attached or associated.The substrate can include any inert substrate or matrix to which nucleicacids can be attached, such as for example glass surfaces, plasticsurfaces, latex, dextran, polystyrene surfaces, polypropylene surfaces,polyacrylamide gels, gold surfaces, and silicon wafers. In someapplications, the substrate is within a channel or other area at aplurality of locations formed in a matrix or array across the samplecontainer 110. System 100 also may include a temperature stationactuator 130 and heater/cooler 135 that can optionally regulate thetemperature of conditions of the fluids within the sample container 110.

In particular implementations, the sample container 110 may beimplemented as a patterned flow cell including a translucent coverplate, a substrate, and a liquid contained therebetween, and abiological sample may be located at an inside surface of the translucentcover plate or an inside surface of the substrate. The flow cell mayinclude a large number (e.g., thousands, millions, or billions) of wellsor regions that are patterned into a defined array (e.g., a hexagonalarray, rectangular array, etc.) into the substrate. Each region may forma cluster (e.g., a monoclonal cluster) of a biological sample such asDNA, RNA, or another genomic material which may be sequenced, forexample, using sequencing by synthesis. The flow cell may be furtherdivided into a number of spaced apart lanes (e.g., eight lanes), eachlane including a hexagonal array of clusters.

Sample container 110 can be mounted on a sample stage 170 to providemovement and alignment of the sample container 110 relative to theobjective lens 142. The sample stage can have one or more actuators toallow it to move in any of three dimensions. For example, in terms ofthe Cartesian coordinate system, actuators can be provided to allow thestage to move in the X, Y and Z directions relative to the objectivelens. This can allow one or more sample locations on sample container110 to be positioned in optical alignment with objective lens 142.Movement of sample stage 170 relative to objective lens 142 can beachieved by moving the sample stage itself, the objective lens, someother component of the imaging system, or any combination of theforegoing. Further implementations may also include moving the entireimaging system over a stationary sample. Alternatively, sample container110 may be fixed during imaging.

In some implementations, a focus (z-axis) component 175 may be includedto control positioning of the optical components relative to the samplecontainer 110 in the focus direction (typically referred to as the zaxis, or z direction). Focus component 175 can include one or moreactuators physically coupled to the optical stage or the sample stage,or both, to move sample container 110 on sample stage 170 relative tothe optical components (e.g., the objective lens 142) to provide properfocusing for the imaging operation. For example, the actuator may bephysically coupled to the respective stage such as, for example, bymechanical, magnetic, fluidic or other attachment or contact directly orindirectly to or with the stage. The one or more actuators can beconfigured to move the stage in the z-direction while maintaining thesample stage in the same plane (e.g., maintaining a level or horizontalattitude, perpendicular to the optical axis). The one or more actuatorscan also be configured to tilt the stage. This can be done, for example,so that sample container 110 can be leveled dynamically to account forany slope in its surfaces.

The structured light emanating from a test sample at a sample locationbeing imaged can be directed through dichroic mirror 160 to one or moredetectors of camera system 140. In some implementations, a filterswitching assembly 165 with one or more emission filters may beincluded, where the one or more emission filters can be used to passthrough particular emission wavelengths and block (or reflect) otheremission wavelengths. For example, the one or more emission filters maybe used to switch between different channels of the imaging system. In aparticular implementation, the emission filters may be implemented asdichroic mirrors that direct emission light of different wavelengths todifferent image sensors of camera system 140.

Camera system 140 can include one or more image sensors to monitor andtrack the imaging (e.g., sequencing) of sample container 110. Camerasystem 140 can be implemented, for example, as a charge-coupled device(CCD) image sensor camera, but other image sensor technologies (e.g.,active pixel sensor) can be used.

Output data (e.g., images) from camera system 140 may be communicated toa real-time SIM imaging component 191 that may be implemented as asoftware application that, as further described below, may reconstructthe images captured during each imaging cycle to create an image havinga higher spatial resolution. The reconstructed images may take intoaccount changes in structure illumination parameters that are predictedover time. In addition, SIM imaging component 191 may be used to trackpredicted SIM parameters and/or make predictions of SIM parameters givenprior estimated and/or predicted SIM parameters.

A controller 195 can be provided to control the operation of structuredillumination imaging system 100, including synchronizing the variousoptical components of system 100. The controller can be implemented tocontrol aspects of system operation such as, for example, configurationof light structuring optical assembly 155 (e.g., selection and/or lineartranslation of diffraction gratings), movement of tube lens 156,focusing, stage movement, and imaging operations. The controller may bealso be implemented to control hardware elements of the system 100 tocorrect for changes in structured illumination parameters over time. Forexample, the controller may be configured to transmit control signals tomotors or other devices controlling a configuration of light structuringoptical assembly 155, motion stage 170, or some other element of system100 to correct or compensate for changes in structured illuminationphase, frequency, and/or orientation over time. In implementations,these signals may be transmitted in accordance with structuredillumination parameters predicted using SIM imaging component 191. Insome implementations, controller 195 may include a memory for storingpredicted and or estimated structured illumination parameterscorresponding to different times and/or sample positions.

In various implementations, the controller 195 can be implemented usinghardware, algorithms (e.g., machine executable instructions), or acombination of the foregoing. For example, in some implementations thecontroller can include one or more CPUs, GPUs, or processors withassociated memory. As another example, the controller can comprisehardware or other circuitry to control the operation, such as a computerprocessor and a non-transitory computer readable medium withmachine-readable instructions stored thereon. For example, thiscircuitry can include one or more of the following: field programmablegate array (FPGA), application specific integrated circuit (ASIC),programmable logic device (PLD), complex programmable logic device(CPLD), a programmable logic array (PLA), programmable array logic (PAL)and other similar processing device or circuitry. As yet anotherexample, the controller can comprise a combination of this circuitrywith one or more processors.

FIG. 3 is an optical diagram illustrating an example opticalconfiguration of a two-arm SIM imaging system 200 that may implementstructured illumination parameter prediction in accordance with someimplementations described herein. The first arm of system 200 includes alight emitter 210A, a first optical collimator 220A to collimate lightoutput by light emitter 210A, a diffraction grating 230A in a firstorientation with respect to the optical axis, a rotating mirror 240A,and a second optical collimator 250A. The second arm of system 200includes a light emitter 210B, a first optical collimator 220B tocollimate light output by light emitter 210B, a diffraction grating 230Bin a second orientation with respect to the optical axis, a rotatingmirror 240B, and a second optical collimator 250B. Although diffractiongratings are illustrated in this example, in other implementations,other beam splitting elements such as a beam splitter cube or plate maybe used to split light received at each arm of SIM imaging system 200.

Each light emitter 210A-210B may be an incoherent light emitter (e.g.,emit light beams output by one or more excitation diodes), or a coherentlight emitter such as emitter of light output by one or more lasers orlaser diodes. In the example of system 200, each light emitter 210A-210Bis an optical fiber that outputs an optical beam that is collimated by arespective collimator 220A-220B.

In some implementations, each optical fiber may be optically coupled toa corresponding light source (not shown) such as a laser. Duringimaging, each optical fiber may be switched on or off using a high-speedshutter (not shown) positioned in the optical path between the fiber andthe light source, or by pulsing the fiber's corresponding light sourceat a predetermined frequency during imaging. In some implementations,each optical fiber may be optically coupled to the same light source. Insuch implementations, a beam splitter or other suitable optical elementmay be used to guide light from the light source into each of theoptical fibers. In such examples, each optical fiber may be switched onor off using a high-speed shutter (not shown) positioned in the opticalpath between the fiber and beam splitter.

In example SIM imaging system 200, the first arm includes a fixedvertical grating 230A to project a grating pattern in a firstorientation (e.g., a vertical fringe pattern) onto the sample, and thesecond arm includes a fixed horizontal grating 230B to project a gratingpattern in a second orientation (e.g., a horizontal fringe pattern) ontothe sample 271. The gratings of SIM imaging system 200 do not need to bemechanically rotated or translated, which may provide improved systemspeed, reliability, and repeatability.

In alternative implementations, gratings 230A and 230B may be mounted onrespective linear motion stages that may be translated to change theoptical path length (and thus the phase) of light emitted by gratings230A and 230B. The axis of motion of linear motion of the stages may beperpendicular or otherwise offset from the orientation of theirrespective grating to realize translation of the grating's pattern alonga sample 271.

Gratings 230A-230B may be transmissive diffraction gratings, including aplurality of diffracting elements (e.g., parallel slits or grooves)formed into a glass substrate or other suitable surface. The gratingsmay be implemented as phase gratings that provide a periodic variationof the refractive index of the grating material. The groove or featurespacing may be chosen to diffract light at suitable angles and tuned tothe minimum resolvable feature size of the imaged samples for operationof SIM imaging system 200. In other implementations, the gratings may bereflective diffraction gratings.

In the example of SIM imaging system 200, the vertical and horizontalpatterns are offset by about 90 degrees. In other implementations, otherorientations of the gratings may be used to create an offset of about 90degrees. For example, the gratings may be oriented such that theyproject images that are offset ±45 degrees from the x or y plane ofsample 271. The configuration of example SIM imaging system 200 may beparticularly advantageous in the case of a regularly patterned sample271 with features on a rectangular grid, as structured resolutionenhancement can be achieved using only two perpendicular gratings (e.g.,vertical grating and horizontal grating).

Gratings 230A-230B, in the example of system 200, are configured todiffract the input beams into a number of orders (e.g., 0 order, ±1orders, ±2 orders, etc.) of which the ±1 orders may be projected on thesample 271. As shown in this example, vertical grating 230A diffracts acollimated light beam into first order diffracted beams (±1 orders),spreading the first orders on the plane of the page, and horizontalgrating 230B diffracts a collimated light beam into first orderdiffracted beams, spreading the orders above and below the plane of thepage (i.e., in a plane perpendicular to the page). To improve efficiencyof the system, the zeroth order beams and all other higher order beams(i.e., ±2 orders or higher) may be blocked (i.e., filtered out of theillumination pattern projected on the sample 271). For example, a beamblocking element (not shown) such as an order filter may be insertedinto the optical path after each diffraction grating to block the0-order beam and the higher order beams. In some implementations,diffraction gratings 230A-230B may configured to diffract the beams intoonly the first orders and the 0-order (undiffracted beam) may be blockedby some beam blocking element.

Each arm includes an optical phase modulator or phase shifter 240A-240Bto phase shift the diffracted light output by each of gratings 230. Forexample, during structured imaging, the optical phase of each diffractedbeam may be shifted by some fraction (e.g., ½, ⅓, ¼, etc.) of the pitch(λ) of each fringe of the structured pattern. In the example of FIG. 3 ,phase modulators 240A and 240B are implemented as rotating windows thatmay use a galvanometer or other rotational actuator to rotate andmodulate the optical path-length of each diffracted beam. For example,window 240A may rotate about the vertical axis to shift the imageprojected by vertical grating 230A on sample 271 left or right, andwindow 240B may rotate about the horizontal axis to shift the imageprojected by horizontal grating 230B on sample 271 up or down.

In other implementations, other phase modulators that change the opticalpath length of the diffracted light (e.g. linear translation stages,wedges, etc.) may be used. Additionally, although optical phasemodulators 240A-240B are illustrated as being placed after gratings230A-230B, in other implementations they may be placed at otherlocations in the illumination system.

In alternative implementations, a single phase modulator may be operatedin two different directions for the different fringe patterns, or asingle phase modulator may use a single motion to adjust both of thepath lengths. For example, a large, rotating optical window may beplaced after mirror 260 with holes 261. In this case, the large windowmay be used in place of windows 240A and 240B to modulate the phases ofboth sets of diffracted beams output by the vertical and horizontaldiffraction gratings. Instead of being parallel with respect to theoptical axis of one of the gratings, the axis of rotation for the largerotating window may be offset 45 degrees (or some other angular offset)from the optical axis of each of the vertical and horizontal gratings toallow for phase shifting along both directions along one common axis ofrotation of the large window. In some implementations, the largerotating window may be replaced by a wedged optic rotating about thenominal beam axis.

In example system 200, a mirror 260 with holes 261 combines the two armsinto the optical path in a lossless manner (e.g., without significantloss of optical power, other than a small absorption in the reflectivecoating). Mirror 260 can be located such that the diffracted orders fromeach of the gratings are spatially resolved, and the unwanted orders canbe blocked. Mirror 260 passes the first orders of light output by thefirst arm through holes 261. Mirror 260 reflects the first orders oflight output by the second arm. As such, the structured illuminationpattern may be switched from a vertical orientation (e.g., grating 230A)to a horizontal orientation (e.g., grating 230B) by turning each emitteron or off or by opening and closing an optical shutter that directs alight source's light through the fiber optic cable. In otherimplementations, the structured illumination pattern may be switched byusing an optical switch to change the arm that illuminates the sample.

Also illustrated in example imaging system 200 are a tube lens 265, asemi-reflective mirror 280, objective 270, and camera 290. For example,tube lens 265 may be implemented to articulate along the z-axis toadjust the structured beam shape and path. Semi-reflective mirror 280may be a dichroic mirror to reflect structured illumination lightreceived from each arm down into objective 270 for projection ontosample 271, and to pass through light emitted by sample 271 (e.g.,fluorescent light, which is emitted at different wavelengths than theexcitation) onto camera 290.

Output data (e.g., images) from camera 290 may be communicated to areal-time SIM imaging component (not shown) that may be implemented as asoftware application that, as further described below, may reconstructthe images captured during each imaging cycle to create an image havinga higher spatial resolution. The reconstructed images may take intoaccount changes in structure illumination parameters that are predictedover time. In addition, the real-time SIM imaging component may be usedto track predicted SIM parameters and/or make predictions of SIMparameters given prior estimated and/or predicted SIM parameters.

A controller (not shown) can be provided to control the operation ofstructured illumination imaging system 200, including synchronizing thevarious optical components of system 200. The controller can beimplemented to control aspects of system operation such as, for example,configuration of each optical arm (e.g., turning on/off each optical armduring capture of phase images, actuation of phase modulators240A-240B), movement of tube lens 265, stage movement (if any stage isused) of sample 271, and imaging operations. The controller may be alsobe implemented to control hardware elements of the system 200 to correctfor changes in structured illumination parameters over time. Forexample, the controller may be configured to transmit control signals todevices (e.g., phase modulators 240A-240B) controlling a configurationof each optical arm or some other element of system 100 to correct orcompensate for changes in structured illumination phase, frequency,and/or orientation over time. As another example, when gratings230A-230B are mounted on linear motion stages (e.g., instead of usingphase modulators 240A-240B), the controller may be configured to controlthe linear motion stages to correct or compensate for phase changes. Inimplementations, these signals may be transmitted in accordance withstructured illumination parameters predicted using a SIM imagingcomponent. In some implementations, the controller may include a memoryfor storing predicted and or estimated structured illuminationparameters corresponding to different times and/or sample positions.

It should be noted that, for the sake of simplicity, optical componentsof SIM imaging system 200 may have been omitted from the foregoingdiscussion. Additionally, although system 200 is illustrated in thisexample as a single channel system, in other implementations, it may beimplemented as a multi-channel system (e.g., by using two differentcameras and light sources that emit in two different wavelengths).

FIG. 4 illustrates simplified illumination fringe patterns that may beprojected onto the plane of a sample 271 by a vertical grating 230A andhorizontal grating 230B of SIM imaging system 200 during one imagingcycle to use structured light to create a high-resolution image. In thisexample, three phase images with a vertical illumination orientation maybe captured using vertical grating 230A, and three phase images with ahorizontal illumination orientation may be captured using horizontalgrating 230B. For each orientation, projected fringes may be phasedshifted in position in steps of ⅓λ (e.g., by setting phase modulator230A or 230B to three different positions) to capture three phase imagesof the orientation pattern.

During capture of each phase image, any light emitted by the sample maybe captured by camera 290. For instance, fluorescent dyes situated atdifferent features of the sample 271 may fluoresce and the resultantlight may be collected by the objective lens 270 and directed to animage sensor of camera 290 to detect the florescence. The captured siximages may be to image an entire sample or a location of a largersample.

Once all images have been captured for the imaging cycle (in thisexample, six images), a high resolution image may be constructed fromthe captured images. For example, a high resolution image may bereconstructed from the six images shown in FIG. 4 . Suitable algorithmsmay be used to combine these various images to synthesize a single imageof the sample with significantly better spatial resolution than any ofthe individual component images.

During construction of the high resolution image, undesired shifts orchanges in structured illumination parameters (e.g., phase, frequency,orientation), may be algorithmically compensated for using structuredillumination parameters predicted in accordance with the disclosure(e.g., predicted changes in phase, frequency, or orientation). Forexample, offsets in the phases, orientation, and/or frequency of thevertical illumination images and/or the horizontal illumination imagesmay be compensated for.

In some implementations, undesired shifts or changes in structuredillumination parameters may be compensated for prior to image capture bycontrolling one or more hardware elements of system 200 to compensatefor those changes in the SIM imaging system. For example, prior to animaging sequence and/or in between capture of images of an imagingsequence, phase drift may be compensated for each optical arm byadjusting a phase shifting element (e.g., rotating mirror, linearactuator, etc.). In some implementations, a combination of hardware andalgorithmic compensation may be implemented.

Although system 200 illustrates a two-arm structured illuminationimaging system that includes two gratings oriented at two differentangles, it should be noted that in other implementations, the technologydescribed herein may be implemented with systems using more than twoarms. In the case of a regularly patterned sample with features on arectangular grid, resolution enhancement can be achieved with only twoperpendicular angles (e.g., vertical grating and horizontal grating) asdescribed above. On the other hand, for image resolution enhancement inall directions for other samples (e.g., hexagonally patterned samples),three grating angles may be used. For example, a three-arm system mayinclude three light emitters and three fixed diffraction gratings (oneper arm), where each diffraction grating is oriented around the opticalaxis of the system to project a respective pattern orientation on thesample (e.g., a 0° pattern, a 120° pattern, or a 240° pattern). In suchsystems, additional mirrors with holes may be used to combine theadditional images of the additional gratings into the system in alossless manner. Alternatively, such systems may utilize one or morepolarizing beam splitters to combine the images of each of the gratings.

FIGS. 5A-5B are schematic diagrams illustrating an example opticalconfiguration of a dual optical grating slide SIM imaging system 500that may implement structured illumination parameter prediction inaccordance with some implementations described herein. In example system500, all changes to the grating pattern projected on sample 570 (e.g.,pattern phase shifts or rotations) may be made by linearly translating amotion stage 530 along a single axis of motion, to select a grating 531or 532 (i.e., select grating orientation) or to phase shift one ofgratings 531-532.

System 500 includes a light emitter 510 (e.g., optical fiber opticallycoupled to a light source), a first optical collimator 520 (e.g.,collimation lens) to collimate light output by light emitter 510, alinear motion stage 530 mounted with a first diffraction grating 531(e.g., horizontal grating) and a second diffraction grating 532 (e.g.vertical grating), a tube lens 540, a semi-reflective mirror 550 (e.g.,dichroic mirror), an objective 560, a sample 570, and a camera 580. Forsimplicity, optical components of SIM imaging system 500 may be omittedfrom FIG. 5A. Additionally, although system 500 is illustrated in thisexample as a single channel system, in other implementations, it may beimplemented as a multi-channel system (e.g., by using two differentcameras and light sources that emit in two different wavelengths).

As illustrated by FIG. 5A, a grating 531 (e.g., a horizontal diffractiongrating) may diffract a collimated light beam into first orderdiffracted light beams (on the plane of the page). As illustrated byFIG. 5B, a diffraction grating 532 (e.g., a vertical diffractiongrating) may diffract a beam into first orders (above and below theplane of the page). In this configuration only a single optical armhaving a single emitter 510 (e.g., optical fiber) and single linearmotion stage is needed to image a sample 570, which may provide systemadvantages such as reducing the number of moving system parts to improvespeed, complexity and cost. Additionally, in system 500, the absence ofa polarizer may provide the previously mentioned advantage of highoptical efficiency. The configuration of example SIM imaging system 200may be particularly advantageous in the case of a regularly patternedsample 570 with features on a rectangular grid, as structured resolutionenhancement can be achieved using only two perpendicular gratings (e.g.,vertical grating and horizontal grating).

To improve efficiency of the system, the zeroth order beams and allother higher order diffraction beams (i.e., ±2 orders or higher) outputby each grating may be blocked (i.e., filtered out of the illuminationpattern projected on the sample 570). For example, a beam blockingelement (not shown) such as an order filter may be inserted into theoptical path after motion stage 530. In some implementations,diffraction gratings 531-532 may configured to diffract the beams intoonly the first orders and the zeroth order (undiffracted beam) may beblocked by some beam blocking element.

In the example of system 500, the two gratings may be arranged about±45° from the axis of motion (or other some other angular offset fromthe axis of motion such as about +40°/−50°, about +30°/−60°, etc.) suchthat a phase shift may be realized for each grating 531-532 along asingle axis of linear motion. In some implementations, the two gratingsmay be combined into one physical optical element. For example, one sideof the physical optical element may have a grating pattern in a firstorientation, and an adjacent side of the physical optical element mayhave a grating pattern in a second orientation orthogonal to the firstorientation.

Single axis linear motion stage 530 may include one or more actuators toallow it to move along the X-axis relative to the sample plane, or alongthe Y-axis relative to the sample plane. During operation, linear motionstage 530 may provide sufficient travel (e.g., about 12-15 mm) andaccuracy (e.g., about less than 0.5 micrometer repeatability) to causeaccurate illumination patterns to be projected for efficient imagereconstruction. In implementations where motion stage 530 is utilized inan automated imaging system such as a fluorescence microscope, it may beconfigured to provide a high speed of operation, minimal vibrationgeneration and a long working lifetime. In implementations, linearmotion stage 530 may include crossed roller bearings, a linear motor, ahigh-accuracy linear encoder, and/or other components. For example,motion stage 530 may be implemented as a high-precision stepper or piezomotion stage that may be translated using a controller.

Output data (e.g., images) from camera 580 may be communicated to areal-time SIM imaging component (not shown) that may be implemented as asoftware application that, as further described below, may reconstructthe images captured during each imaging cycle to create an image havinga higher spatial resolution. The reconstructed images may take intoaccount changes in structure illumination parameters that are predictedover time. In addition, the real-time SIM imaging component may be usedto track predicted SIM parameters and/or make predictions of SIMparameters given prior estimated and/or predicted SIM parameters.

A controller (not shown) can be provided to control the operation ofstructured illumination imaging system 500, including synchronizing thevarious optical components of system 500. The controller can beimplemented to control aspects of system operation such as, for example,translation of linear motion stage 530, movement of tube lens 540, stagemovement (if any stage is used) of sample 570, and imaging operations.The controller may be also be implemented to control hardware elementsof the system 500 to correct for changes in structured illuminationparameters over time. For example, the controller may be configured totransmit control signals to devices (e.g., linear motion stage 530) tocorrect or compensate for changes in structured illumination phase,frequency, and/or orientation over time. In implementations, thesesignals may be transmitted in accordance with structured illuminationparameters predicted using a SIM imaging component. In someimplementations, the controller may include a memory for storingpredicted and or estimated structured illumination parameterscorresponding to different times and/or sample positions.

Although the example of FIGS. 5A-5B illustrates a dual optical gratingslide imaging system that may implement structured illuminationparameter prediction, structured illumination parameter prediction maybe implemented in SIM imaging systems that use a linear motion actuatormounted with more than two diffraction gratings.

FIG. 6 illustrates simplified illumination fringe patterns that may beprojected onto the plane of a sample 570 by a first diffraction gratingand a second diffraction grating of a dual optical grating slide SIMimaging system 500 during image capture for a structured illuminationimaging cycle. For example, a SIM imaging system 500 may use a firstdiffraction grating 531 and second diffraction grating 532 to generatethe illumination patterns shown in FIG. 6 . As illustrated in theexample of FIG. 6 , the two gratings project perpendicular fringepatterns on the surface of sample 570 and are arranged about ±45° fromthe axis of motion of linear motion stage 530.

For example, a first grating (e.g., grating 531), may projectfirst-order illumination fringes on sample 570. Any light emitted by thesample may be captured by camera 580 and a first phase image of thefirst pattern (e.g., +45° pattern) may be captured to create a firstphase image. To capture additional phase shifted images, the patternprojected by the grating may be phase shifted by translating the linearmotion stage. These phase shift motions are illustrated as steps 1 and 2in FIG. 6 . The phase shift motions may provide small (e.g., about 3 to5 micrometers or smaller) moves of the gratings to slightly shift thefringe pattern projected on the grating.

Following capture of all phase shifted images fora diffraction grating,system 500 may switch diffraction gratings by translating the linearmotion stage 530 to optically couple another diffraction grating to thelight source of the imaging system (e.g., transition from FIG. 5A to5B). This motion is illustrated as step 3 in the example of FIG. 6 . Inthe case of diffraction grating changes, the linear motion stage mayprovide a relatively large translation (e.g., on the order of 12-15 mm).

A series of phase images may then be captured for the next grating. Forinstance, as illustrated by FIG. 6 , a second diffraction grating mayproject first-order illumination fringes on the sample, and theprojected fringes may be shifted in position by translating the linearmotion stage 530 to capture three phase images of the grating's pattern(e.g., steps 4 and 5 of FIG. 6 ).

Once all images have been captured for the imaging cycle (in thisexample, six images), a high resolution image may be constructed fromthe captured images. For example, a high resolution image may bereconstructed from the six images shown in FIG. 6 . Suitable algorithmsmay be used to combine these various images to synthesize a single imageof the sample with significantly better spatial resolution than any ofthe individual component images.

During construction of the high resolution image, undesired shifts orchanges in structured illumination parameters (e.g., phase, frequency,orientation), may be algorithmically compensated for using structuredillumination parameters predicted in accordance with the disclosure(e.g., predicted changes in phase, frequency, or orientation). Forexample, offsets in the phases, orientation, and/or frequency of thevertical illumination images and/or the horizontal illumination imagesmay be compensated for.

In some implementations, undesired shifts or changes in structuredillumination parameters may be compensated for prior to image capture bycontrolling one or more hardware elements of system 500 to compensatefor those changes in the SIM imaging system. For example, prior to animaging sequence and/or in between capture of images of an imagingsequence, phase drift may be compensated for by translating linearmotion stage 530. In some implementations, a combination of hardware andalgorithmic compensation may be implemented.

In accordance with implementations described herein, structuredillumination parameters may be predicted for a particular point in timeusing estimates of structured illumination parameters obtained fromimages captured before and/or after that point in time. For example,computational resource limitations may limit the rate at which a SIMimaging system (e.g., system 100, 200, or 500) may directly estimatestructured illumination parameters such as phase, frequency, and/ororientation from captured images. In some cases, a SIM imaging systemmay directly estimate or measure a structured illumination parameterevery phase image, in which case it may not be necessary to predictstructured illumination parameters. However, in other cases, a SIMimaging system may only be able to directly estimate or measure astructured illumination parameter for some phase images of an imagingcycle, once per imaging cycle, or even less frequently (e.g., every 3,5, 10, 50, or 100 imaging cycles). In such cases, to keep up with theimage sampling rate of the system, it may be advantageous to leverage adirect estimate of the structured illumination parameter that wasobtained for a particular point in time and/or space to make predictionsabout the structured illumination parameter at other points in timeand/or space.

To mathematically illustrate one example of this principal, correlationagainst a reference is one way to estimate structured illuminationparameters.Correlation Output=Σ_(r) c(x)h(x−f),  (1)where h(x) is a reference which may either be known or derived fromimage data, c(x) is derived from image data which is correlated to thereference, and f is a value to be estimated (in this example,frequency). It should be noted that other alternative estimationtechniques may be utilized in accordance with the disclosure.

In the example of Equation (1), one correlation output may be generatedfor each of a number of hypothetical values of f. The parameter estimatef may be obtained as the value of f which maximizes the magnitude of thecorrelation. However, in many cases, a large number of hypotheticalvalues of f may need to be attempted in order to maximize thecorrelation output. The large search space may increase thecomputational requirements, and as a result, may cause reduced systemthroughput (i.e., less data processed per unit of time).

To circumvent this problem, information from a prior f estimate may beused to ascertain a “neighborhood” of a new f value to be determined. Asan example, consider FIG. 7 , which shows an estimate ϕ which varies inspace (X) and time (T). As illustrated by FIG. 7 , an initial value forϕ may be obtained for the X and T coordinates corresponding to the Ablock. Assuming that the value of the estimate varies slowly in space ortime, the estimate from the A block (ϕ_(A)) may be used as an initialvalue for either the B or E blocks. More specifically, the search spacefor the B and E blocks may be restricted to values in the “neighborhood”of the value of ϕ obtained from the A block. With this approach, thetime required to identify ϕ may be reduced considerably, and as aresult, the amount of data processed in a unit of time may be increasedaccordingly.

To extend this concept, a trend of the estimate variation may bepredicted in either the space (X) or time (T) dimension. As an example,consider FIG. 7 , where the per block estimate increases by Δ_(ϕx) inthe spatial dimension, and by Δ_(ϕT) in the time dimension. Given thisobservation, an initial estimate for Block B could be derived asϕ_(A)+Δ_(ϕx), as illustrated by FIG. 8 . Further, an initial estimate ofBlock E could be derived as ϕ_(A)+Δ_(ϕT). Other predictors in both the Xand T dimensions using values from multiple blocks could also beimplemented.

FIG. 9 is an operational flow diagram illustrating an exampleinterpolation method 900 for predicting structured illumination usingestimates of structured illumination parameters obtained from multipleimages captured by a structured illumination system. In implementations,method 700 may be implemented by executing machine readable instructionsstored in a memory of a SIM imaging system (e.g., system 100, 200, or500).

At operation 910, a first SIM image sample may be obtained. For example,a phase image of a sample may be captured at a first point in time. Atoperation 920, a structured illumination parameter may be estimatedusing the captured first image. For example, any one of a structuredillumination phase, frequency, orientation, or modulation order may beestimated. The estimate may be obtained at particular point in time,space, and/or temperature.

At operation 930 a second SIM image sample may be obtained. For example,a phase image of the sample may be captured at a second point in timeafter a first point in time when the first SIM image sample is captured.In some implementations, the first image of the sample and the secondimage of the sample may be captured during the same imaging sequence(e.g., as part of an imaging sequence that generates six phase images ornine phase images that are constructed into a higher resolution image).In other implementations, the first image and the second image may becaptured during different imaging sequences. At operation 940, astructured illumination parameter may be estimated using the capturedsecond image. For example, any one of a structured illumination phase,frequency, orientation, or modulation order may be estimated. Theestimate may be obtained at particular point in time, space, and/ortemperature.

At operation 950, using at least the estimate of the structuredillumination parameter from the first image and the estimate of thestructured illumination from the second image, a structured illuminationparameter corresponding to a third image may be predicted, where thethird image is at a point in time, space (e.g., sample position), and/ortemperature (e.g., sample temperature) between the first image and thesecond image. For example, the third image may have been captured afterthe first image but before the second image. As another example, thethird image may be captured at a later time at a position between thefirst image and the second image.

In some implementations, this prediction may be based on at least adetermined rate of change of the structured illumination parameterbetween two points in time. By way of mathematical illustration, for afirst time T1, and a second time T2, if it is determined that astructured illumination phase has drifted by an amount Δ_(ϕT), then therate of change (e.g., drift) of the phase may be expressed asΔ_(ϕT)/(T2−T1). Using interpolation, the amount of phase drift for atime T3 may be predicted. For example, if the phase drifted from a 5degree offset at time T1 to a 15 degree offset at time T2, then it maybe predicted that the phase had drifted to a 10 degree offset at a timeT3 halfway between these two times.

Although method 900 is primarily described in the context of applyinginterpolation to predict a structured illumination parameter at aparticular point in time, space, and/or temperature given two knownestimates of the structured illumination parameter, it should be notedthat method 900 may be extended to the case where there are more thantwo known estimates. In such cases, an appropriate trend estimationfunction may be used to predict the structured illumination parameter.For example, in the case of linear trend estimation, a least-squares fitmay be applied to the known estimates to interpolate and predict thestructured illumination parameter at a particular point in time, space,and/or temperature. In some implementations, a prediction of astructured illumination parameter may be updated over time as additionalestimates are gathered. Additionally, although method 900 is describedas using estimates from the first image and the second image to predicta parameter corresponding to a third image, in some implementations,only one of the two estimates may be used (e.g., by holding theestimate) to predict the parameter.

Additionally, although method 900 is described in the context ofapplying interpolation to predict a structured illumination parameter ata particular time given two known estimates of the structuredillumination parameter at different times, method 900 may also beextended to consider dimensions of space (e.g., a location or subset ofan imaged sample) and temperature. In some instances, a joint predictionthat considers multiple parameters (e.g., space, time, and/ortemperature) may be applied. For example, as illustrated by FIG. 7 , atrend in both time and space may be considered in predicting structuredillumination parameters. Alternatively, a trend in the structuredillumination parameter in just space may be considered.

FIG. 10 is an operational flow diagram illustrating an exampleextrapolation method 1000 for predicting structured illuminationparameters using estimates of structured illumination parametersobtained from two or more images. In implementations, method 700 may beimplemented by executing machine readable instructions stored in amemory of a SIM imaging system (e.g., system 100, 200, or 500).

Operations 910-940 of method 1000 may be performed as discussed abovewith reference to method 900. For example, a structured illuminationfrequency may be estimated at a first point in time and second point intime using captured images.

At operation 1050, using at least the estimate of the structuredillumination parameter from the first image and the estimate of thestructured illumination from the second image, a structured illuminationparameter corresponding to a third image may be predicted, where thethird image is at a point in time, space (e.g., sample position), and/ortemperature (e.g., sample temperature) after both the first image andthe second image, or before both the first image and the second image.In some implementations, this prediction may be based on at least adetermined rate of change of the structured illumination parameterbetween the two points in time. By way of mathematical illustration, fora first time T1, and a second time T2, if it is determined that astructured illumination frequency has drifted by an amount Δf, then therate of change (e.g., drift) of the phase may be expressed asΔf/(T2−T1). Using extrapolation, the total amount of frequency drift ata later time T3 may be predicted.

Although method 1000 is described in the context of applyingextrapolation to predict a structured illumination parameter at aparticular point in time, space, and/or temperature given two knownestimates of the structured illumination parameter, it should be notedthat as in the case of method 900, method 1000 may be extended to thecase where there are more than two known estimates. In such cases, anappropriate trend estimation function may be used to predict thestructured illumination parameter. For example, in the case of lineartrend estimation, a least-squares fit may be applied to the knownestimates to extrapolate and predict the structured illuminationparameter. In some implementations, a prediction of a structuredillumination parameter may be updated over time as additional estimatesare gathered.

Additionally, although method 1000 is described as using estimates fromthe first image and the second image to predict a parametercorresponding to a third image, in some implementations, only one of thetwo estimates may be used (e.g., by holding the estimate) to predict theparameter.

Additionally, although method 1000 is described in the context ofapplying extrapolation to predict a structured illumination parameter ata particular time given two known estimates of the structuredillumination parameter at different times, as in the case of method 900,method 1000 may also be extended to consider other dimensions such asspace and temperature.

In implementations of methods 900 and 1000, the structured illuminationparameter estimated using the first image, the structured illuminationparameter estimated using the second image, and/or the structuredillumination parameter predicted for the third image may be stored in amemory of the SIM imaging system. For instance, the estimated/predictedparameters may be stored in a history file such as a lookup table to bereferenced during high resolution image construction, during adjustmentsof SIM imaging system hardware components to compensate for structuredillumination parameter changes, and/or to facilitate prediction of otherstructured illumination parameters at other points in time, space,and/or temperature. In implementations, the time, sample location, andsample temperature corresponding to each estimation or prediction may bestored.

In implementations of methods 900 and 1000, the first and secondestimates used to predict the structured illumination parameter may begenerated using a plurality of images. As such, one or more images froma first set of images (e.g., 1, 2, 3, 4, 5, 6, etc.) in an imagingsequence may be used to generate a first estimate, and one or moreimages from a second set of images (e.g., 1, 2, 3, 4, 5) in an imagingsequence may be used to generate a second estimate.

FIG. 11 is an operational flow diagram illustrating an example method1100 of using a predicted structured illumination parameter during highresolution image reconstruction to compensate for undesired changes instructured illumination parameters over time. In implementations, method1100 may be implemented by executing machine readable instructionsstored in a memory of a SIM imaging system (e.g., system 100, 200, or500).

At operation 1110, a structured illumination parameter may be predictedfor a captured image (e.g., a phase image) using an interpolationmethod. For example, a structured illumination parameter may bepredicted at a point in time corresponding to the captured image byimplementing method 900. At operation 1120, a high resolution imageconstruction may be performed using the captured image (e.g., phaseimage) and other captured images (e.g., other captured phase images).During high resolution image reconstruction, the predicted structuredillumination parameter may be used to compensate for changes in thestructured illumination parameter over a dimension of time, space,and/or temperature. For example, changes in frequency, phase, and/ororientation may be compensated for. In some cases, operation 1120 maycomprise using multiple predicted structured illumination parameters.For example, structured illumination parameters may be predicted formore than one phase image. Additionally, two or more of phase,frequency, and orientation may be predicted for a given phase image.

FIG. 12 is an operational flow diagram illustrating an example method1200 of using a predicted structured illumination parameter adjustmentsof SIM imaging system hardware components to compensate for structuredillumination parameter changes over time. At operation 1210, astructured illumination parameter may be predicted using anextrapolation method. For example, a structured illumination parametermay be predicted at a future point in time by implementing method 1000.

At operation 1220, a mechanical and/or optical component of the SIMimaging device may be adjusted using at least the predicted structuredillumination parameter. For instance, based on a predicted phase driftat a point in time T, a hardware component of the SIM imaging system maybe adjusted prior to phase image capture at time T.

For example, one or more components of light structuring opticalassembly 155 may be adjusted to compensate for phase and/or orientationchanges that are predicted for an upcoming time for SIM imaging system100. As another example, a rotating mirror 240A or 240B may be adjustedto compensate for phase changes that are predicted for an upcoming timefor SIM imaging system 200. As a further example, a linear translationstage 530 be translated to compensate for phase changes that arepredicted for an upcoming for SIM imaging system 500. As a furtherexample, orientation changes that are predicted for a SIM imaging systemmay be compensated by adjusting one or more of a translation stagecarrying a sample and an optical path from a light source to the sample.

In some implementations, the techniques described herein for structuredillumination parameter prediction may be applied to a single capturedimage sample by dividing the captured image sample into a plurality ofimage subsections. For example, in some implementations, a method mayinclude: obtaining an image sample; dividing the image sample into aplurality of image subsections (e.g., three or more subsections);estimating a first structured illumination parameter using a first imagesubsection of the plurality of image subsections; estimating a secondstructured illumination parameter using a second image subsection of theplurality of image subsections; and using at least the estimate of thestructured illumination parameter from the first image subsection andthe estimate of the structured illumination parameter from the secondimage subsection, predicting a structured illumination parametercorresponding to a third image subsection of the plurality of imagesubsections. The structured illumination parameter that is predicted forthe third image subsection may be any one of a structured illuminationphase, frequency, orientation, or modulation order. In someimplementations, structured illumination parameters obtained from morethan two image subsections may be used to predict a structuredillumination parameter for another image subsection. For example, atrend estimation function or other appropriate fitting function may beapplied to the known estimates from the other image subsections topredict the structured illumination parameter for another imagesubsection. In other implementations, an estimate of the structuredillumination parameter obtained from a first image subsection may beused as the predicted structured illumination parameter for a secondimage subsection.

Applying the interpolation techniques described herein, the third imagesubsection may lie at a point in space (e.g., sample position) ortemperature (e.g., sample temperature) between the first imagesubsection and the second image subsection. For example, the third imagesubsection may lie between the first image subsection and the secondimage subsection along a cartesian axis. In two-dimensional cartesianspace, subsections may be defined by a grid that divides an image intorectangles having equal area, though alternative definitions of asubsection are possible. As another example, the third image subsectionmay be at a sample temperature that is greater than a sample temperatureof the first image subsection but lower than a sample temperature of thesecond image subsection.

Applying the extrapolation techniques described herein, the third imagesubsection may lie at a point in space (e.g., sample position) ortemperature (e.g., sample temperature) that is after or before the firstimage subsection and the second image subsection. For example, the thirdimage subsection may lie after both the first image subsection and thesecond image subsection along a cartesian axis. As another example, thethird image subsection may be at a sample temperature lower than asample temperature of the first image subsection and lower than a sampletemperature of the second image subsection.

In implementations, these techniques for using subsections of an imageto predict structured illumination parameter for other subsection(s) ofthe image may be used in combination with the techniques describedherein for using structured illumination parameters estimated from oneor more images to predict structured illumination parameters for anotherimage.

As used herein, the term component might describe a given unit offunctionality that can be performed in accordance with one or moreimplementations of the present application. As used herein, a componentmight be implemented utilizing any form of hardware, software, or acombination thereof. For example, one or more processors, controllers,FPGAs, CPUs, GPUs, ASICs, PLAs, PALs, CPLDs, logical components,software routines or other mechanisms might be implemented to make up acomponent. In implementation, the various components described hereinmight be implemented as discrete components or the functions andfeatures described can be shared in part or in total among one or morecomponents. In other words, as would be apparent to one of ordinaryskill in the art after reading this description, the various featuresand functionality described herein may be implemented in any givenapplication and can be implemented in one or more separate or sharedcomponents in various combinations and permutations. Even though variousfeatures or elements of functionality may be individually described orclaimed as separate components, one of ordinary skill in the art willunderstand that these features and functionality can be shared among oneor more common software and hardware elements, and such descriptionshall not require or imply that separate hardware or software componentsare used to implement such features or functionality.

FIG. 13 illustrates an example computing component 1300 that may be usedto implement various features of the methods disclosed herein. Computingcomponent 1300 may represent, for example, computing or processingcapabilities found within imaging devices; desktops and laptops;hand-held computing devices (tablets, smartphones, etc.); mainframes,supercomputers, workstations or servers; or any other type ofspecial-purpose or general-purpose computing devices as may be desirableor appropriate for a given application or environment. Computingcomponent 1300 might also represent computing capabilities embeddedwithin or otherwise available to a given device. As used herein, theterm “computing device” may refer to hardware of a computing component.

Computing component 1300 might include, for example, one or moreprocessors, controllers, control components, or other processingdevices, such as a processor 1304. Processor 1304 might be implementedusing a general-purpose or special-purpose processing engine such as,for example, a microprocessor, controller, or other control logic.Processor 1304 may be a type of computing device. In the illustratedexample, processor 1304 is connected to a bus 1302, although anycommunication medium can be used to facilitate interaction with othercomponents of computing component 1300 or to communicate externally.

Computing component 1300 might also include one or more memorycomponents, simply referred to herein as main memory 1308. For example,preferably random access memory (RAM) or other dynamic memory, might beused for storing information and instructions to be executed byprocessor 1304. Main memory 1308 might also be used for storingtemporary variables or other intermediate information during executionof instructions to be executed by processor 1304. Computing component1300 might likewise include a read only memory (“ROM”) or other staticstorage device coupled to bus 1302 for storing static information andinstructions for processor 1304.

The computing component 1300 might also include one or more variousforms of information storage mechanism 1310, which might include, forexample, a media drive 1312 and a storage unit interface 1320. The mediadrive 1312 might include a drive or other mechanism to support fixed orremovable storage media 1314. For example, a hard disk drive, a solidstate drive, an optical disk drive, a CD, DVD, or BLU-RAY drive (R orRW), or other removable or fixed media drive might be provided.Accordingly, storage media 1314 might include, for example, a hard disk,a solid state drive, cartridge, optical disk, a CD, a DVD, a BLU-RAY, orother fixed or removable medium that is read by, written to or accessedby media drive 1312. As these examples illustrate, the storage media1314 can include a computer usable storage medium having stored thereincomputer software or data.

In alternative embodiments, information storage mechanism 1310 mightinclude other similar instrumentalities for allowing computer programsor other instructions or data to be loaded into computing component1300. Such instrumentalities might include, for example, a fixed orremovable storage unit 1322 and an interface 1320. Examples of suchstorage units 1322 and interfaces 1320 can include a program cartridgeand cartridge interface, a removable memory (for example, a flash memoryor other removable memory component) and memory slot, a PCMCIA slot andcard, and other fixed or removable storage units 1322 and interfaces1320 that allow software and data to be transferred from the storageunit 1322 to computing component 1300.

Computing component 1300 might also include a communications interface1324. Communications interface 1324 might be used to allow software anddata to be transferred between computing component 1300 and externaldevices. Examples of communications interface 1324 might include aperipheral interface such as the Peripheral Component InterconnectExpress (PCIe) interface, a modem or softmodem, a network interface(such as an Ethernet, network interface card, WiMedia, IEEE 802.XX orother interface), a BLUETOOTH interface, a communications port (such asfor example, a USB port, USB-C port, THUNDERBOLT port, or other port),or other communications interface. Software and data transferred viacommunications interface 1324 might typically be carried on signals,which can be electronic, electromagnetic (which includes optical) orother signals capable of being exchanged by a given communicationsinterface 1324. These signals might be provided to communicationsinterface 1324 via a channel 1328. This channel 1328 might carry signalsand might be implemented using a wired or wireless communication medium.Some examples of a channel might include a phone line, a cellular link,an RF link, an optical link, a network interface, a local or wide areanetwork, and other wired or wireless communications channels.

In this document, the terms “computer readable medium”, “computer usablemedium” and “computer program medium” are used to generally refer tonon-transitory mediums, volatile or non-volatile, such as, for example,memory 1308, storage unit 1322, and media 1314. These and other variousforms of computer program media or computer usable media may be involvedin carrying one or more sequences of one or more instructions to aprocessor for execution. Such instructions embodied on the medium, aregenerally referred to as “computer program code” or a “computer programproduct” (which may be grouped in the form of computer programs or othergroupings). When executed, such instructions might enable the computingcomponent 1300 to perform features or functions of the presentapplication as discussed herein.

Although described above in terms of various exemplary embodiments andimplementations, it should be understood that the various features,aspects and functionality described in one or more of the individualembodiments are not limited in their applicability to the particularembodiment with which they are described, but instead can be applied,alone or in various combinations, to one or more of the otherembodiments of the application, whether or not such embodiments aredescribed and whether or not such features are presented as being a partof a described embodiment. Thus, the breadth and scope of the presentapplication should not be limited by any of the above-describedexemplary embodiments.

Terms and phrases used in this document, and variations thereof, unlessotherwise expressly stated, should be construed as open ended as opposedto limiting. As examples of the foregoing: the term “including” shouldbe read as meaning “including, without limitation” or the like; the term“example” is used to provide exemplary instances of the item indiscussion, not an exhaustive or limiting list thereof; the terms “a” or“an” should be read as meaning “at least one,” “one or more” or thelike; and adjectives such as “conventional,” “traditional,” “normal,”“standard,” “known” and terms of similar meaning should not be construedas limiting the item described to a given time period or to an itemavailable as of a given time, but instead should be read to encompassconventional, traditional, normal, or standard technologies that may beavailable or known now or at any time in the future. Likewise, wherethis document refers to technologies that would be apparent or known toone of ordinary skill in the art, such technologies encompass thoseapparent or known to the skilled artisan now or at any time in thefuture.

The presence of broadening words and phrases such as “one or more,” “atleast,” “but not limited to” or other like phrases in some instancesshall not be read to mean that the narrower case is intended or requiredin instances where such broadening phrases may be absent.

Additionally, the various embodiments set forth herein are described interms of exemplary block diagrams, flow charts and other illustrations.As will become apparent to one of ordinary skill in the art afterreading this document, the illustrated embodiments and their variousalternatives can be implemented without confinement to the illustratedexamples. For example, block diagrams and their accompanying descriptionshould not be construed as mandating a particular architecture orconfiguration.

The terms “substantially” and “about” used throughout this disclosure,including the claims, are used to describe and account for smallfluctuations, such as due to variations in processing. For example, theycan refer to less than or equal to ±5%, such as less than or equal to±2%, such as less than or equal to ±1%, such as less than or equal to±0.5%, such as less than or equal to ±0.2%, such as less than or equalto ±0.1%, such as less than or equal to ±0.05%.

To the extent applicable, the terms “first,” “second,” “third,” etc.herein are merely employed to show the respective objects described bythese terms as separate entities and are not meant to connote a sense ofchronological order, unless stated explicitly otherwise herein.

While various embodiments of the present disclosure have been describedabove, it should be understood that they have been presented by way ofexample only, and not of limitation. Likewise, the various diagrams maydepict an example architectural or other configuration for thedisclosure, which is done to aid in understanding the features andfunctionality that can be included in the disclosure. The disclosure isnot restricted to the illustrated example architectures orconfigurations, but the desired features can be implemented using avariety of alternative architectures and configurations. Indeed, it willbe apparent to one of skill in the art how alternative functional,logical or physical partitioning and configurations can be implementedto implement the desired features of the present disclosure. Also, amultitude of different constituent module names other than thosedepicted herein can be applied to the various partitions. Additionally,with regard to flow diagrams, operational descriptions and methodclaims, the order in which the steps are presented herein shall notmandate that various embodiments be implemented to perform the recitedfunctionality in the same order unless the context dictates otherwise.

Although the disclosure is described above in terms of various exemplaryembodiments and implementations, it should be understood that thevarious features, aspects and functionality described in one or more ofthe individual embodiments are not limited in their applicability to theparticular embodiment with which they are described, but instead can beapplied, alone or in various combinations, to one or more of the otherembodiments of the disclosure, whether or not such embodiments aredescribed and whether or not such features are presented as being a partof a described embodiment. Thus, the breadth and scope of the presentdisclosure should not be limited by any of the above-described exemplaryembodiments.

It should be appreciated that all combinations of the foregoing concepts(provided such concepts are not mutually inconsistent) are contemplatedas being part of the inventive subject matter disclosed herein. Inparticular, all combinations of claimed subject matter appearing in thisdisclosure are contemplated as being part of the inventive subjectmatter disclosed herein.

What is claimed is:
 1. An imaging device, comprising: a light emitter toemit light; a beam splitter to split light emitted by the light emitterto project a structured illumination pattern on a sample; a controllerconfigured to cause the imaging device to perform operations comprising:capturing a first image of the sample at a first time, a first sampleposition, or a first sample temperature; estimating, using at least thecaptured first image, a first structured illumination parameter;capturing a second image of the sample at a second time, a second sampleposition, or a second sample temperature; estimating, using at least thecaptured second image, a second structured illumination parameter; andpredicting, using at least the first structured illumination parameteror the second structured illumination parameter, a third structuredillumination parameter corresponding to a third image of the samplecaptured at a third time, a third sample position, or a third sampletemperature, wherein each of the first, second, and third structuredillumination parameters comprises a phase, a frequency, an orientation,or a modulation order.
 2. The imaging device of claim 1, wherein: thefirst image is captured at the first time; the second image is capturedat the second time after the first time; the third image is captured atthe third time between the first time and the second time; andpredicting the third structured illumination parameter comprises:determining a rate of change from the first structured illuminationparameter at the first time to the second structured illumination at thesecond time; and predicting, using at least the determined rate ofchange, the third structured illumination parameter at the third time.3. The imaging device of claim 1, wherein: the first image is capturedat the first time; the second image is captured at the second time afterthe first time; the third image is captured at the third time after boththe first time and the second time; and predicting the third structuredillumination parameter comprises: predicting, using at least the firststructured illumination parameter and the second structured illuminationparameter, the third structured illumination parameter at the thirdtime.
 4. The imaging device of claim 3, wherein: the imaging devicefurther comprises an optical phase modulator or phase shifter that ispositioned after the beam splitter in an optical path of the imagingdevice; and the operations further comprise: prior to capturing thethird image at the third time, adjusting, using at least the predictedthird structured illumination parameter, the optical phase modulator orphase shifter to adjust a phase or orientation of the structuredillumination pattern.
 5. The imaging device of claim 4, wherein theoptical phase modulator or phase shifter is a rotating mirror; and thebeam splitter comprises a diffraction grating.
 6. The imaging device ofclaim 3, wherein: the beam splitter comprises a diffraction grating; theimaging device further comprises a translation stage carrying thediffraction grating; and the operations further comprise: prior tocapturing the third image at the third time, adjusting, using at leastthe predicted third structured illumination parameter, the translationstage to adjust a phase or orientation of the structured illuminationpattern.
 7. The imaging device of claim 3, wherein: the imaging devicefurther comprises a sample translation stage carrying the sample; andthe operations further comprise: prior to capturing the third image atthe third time, adjusting, using at least the predicted third structuredillumination parameter, the sample translation stage to adjust a phaseor orientation of the structured illumination pattern.
 8. The imagingdevice of claim 3, wherein: the controller comprises a memory; and theoperations further comprise: storing in the memory: the first structuredillumination parameter, the second structured illumination parameter,and the third structured illumination parameter; and reducing, using oneor more of the stored first structured illumination, the stored secondstructured illumination parameter, the stored third structuredillumination parameter, or a stored value based on known physicalcharacteristics of the imaging device, a search space for a fourthstructured illumination parameter for a fourth image.
 9. The imagingdevice of claim 1, wherein: the first image of the sample is captured atthe first sample temperature; the first structured illuminationparameter is estimated at the first sample temperature; the second imageof the sample is captured at the second sample temperature; the secondstructured illumination parameter is estimated at the second sampletemperature; and predicting the third structured illumination parameter,comprises: predicting the third structured illumination parameter at thethird sample temperature.
 10. The imaging device of claim 1, wherein theoperations further comprise: dividing the first image of the sample intoa plurality of image subsections; estimating, at the computing device,using at least a first image subsection of the plurality of imagesubsections, a fourth structured illumination parameter; estimating, atthe computing device, using at least a second image subsection of theplurality of image subsections, a fifth structured illuminationparameter; and predicting, at the computing device, using at least thefourth structured illumination parameter or the fifth structuredillumination parameter, a sixth structured illumination parametercorresponding to a third image subsection of the plurality of imagesubsections.
 11. The imaging device of claim 1, wherein the operationsfurther comprise: dividing the first image of the sample into aplurality of image subsections; estimating, at the computing device,using at least a first image subsection of the plurality of imagesubsections, a fourth structured illumination parameter; and using theestimated fourth structured illumination parameter as a predictedstructured illumination parameter of a second image subsection of theplurality of image subsections.
 12. An imaging device, comprising: alight emitter to emit light; a diffraction grating to split lightemitted by the light emitter to project a structured illuminationpattern on a sample; a processor; and a non-transitory computer-readablemedium having executable instructions stored thereon that, when executedby the processor, cause the imaging device to perform operationscomprising: capturing a first image of the sample; estimating, using atleast the captured first image, a first structured illuminationparameter; capturing a second image of the sample; estimating, using atleast the captured second image, a second structured illuminationparameter; and predicting, using at least the first structuredillumination parameter and the second structured illumination parameter,a third structured illumination parameter corresponding to a third imageof the sample, wherein each of the first, second, and third structuredillumination parameters comprises a phase, a frequency, an orientation,or a modulation order.
 13. The imaging device of claim 12, wherein: thefirst image is captured at a first time; the second image is captured ata second time after the first time; and the third image is captured at athird time between the first time and the second time.
 14. The imagingdevice of claim 12, wherein: the first image is captured at a firsttime; the second image is captured at a second time after the firsttime; the third image is captured at a third time after both the firsttime and the second time; and predicting the third structuredillumination parameter comprises: predicting, using at least the firststructured illumination parameter and the second structured illuminationparameter, the third structured illumination parameter at the thirdtime.
 15. The imaging device of claim 14, wherein: the imaging devicefurther comprises an optical phase modulator or phase shifter that ispositioned after the diffraction grating in an optical path of theimaging device; and the operations further comprise: prior to capturingthe third image at the third time, adjusting, using at least thepredicted third structured illumination parameter, the optical phasemodulator or phase shifter to adjust a phase or orientation of thestructured illumination pattern.
 16. The imaging device of claim 14,wherein: the imaging device further comprises a translation stagecarrying the diffraction grating; and the operations further comprise:prior to capturing the third image at the third time, adjusting, usingat least the predicted third structured illumination parameter, thetranslation stage to adjust a phase or orientation of the structuredillumination pattern.
 17. The imaging device of claim 16, wherein: thediffraction grating is a first diffraction grating of the imagingdevice; and the imaging device further comprises a second diffractiongrating carried by the translation stage.
 18. The imaging device ofclaim 14, wherein: the imaging device further comprises a sampletranslation stage carrying the sample; and the operations furthercomprise: prior to capturing the third image at the third time,adjusting, using at least the predicted third structured illuminationparameter, the sample translation stage to adjust a phase or orientationof the structured illumination pattern.
 19. The imaging device of claim12, wherein: predicting the third structured illumination parametercomprises: applying a least-squares fit to at least the first structuredillumination parameter and the second structured illumination parameter.20. An imaging device, comprising: a light emitter to emit light; adiffraction grating to split light emitted by the light emitter toproject a structured illumination pattern on a sample; a processor; anda non-transitory computer-readable medium having executable instructionsstored thereon that, when executed by the processor, cause the imagingdevice to perform operations comprising: capturing a first image of thesample; estimating, using at least the captured first image, a firststructured illumination parameter; capturing a second image of thesample; estimating, using at least the captured second image, a secondstructured illumination parameter; predicting, using at least the firststructured illumination parameter and the second structured illuminationparameter, a third structured illumination parameter corresponding to athird image of the sample; and prior to capturing the third image of thesample, adjusting, based on the predicted third structured illuminationparameter, the structured illumination pattern.